IMPORTING MODULES AND VARIABLE DEFINITION

Import Statements for importing all the module and the python files which we have created for our project.

Define a variable for image size for the image which we will be giving to the model (after preprocessing)

Define a variable for the total number of classes (Traffic Signs) that we have to classify.

In [1]:
from keras.utils import np_utils
import numpy as np
from random import randint
from keras.callbacks import ModelCheckpoint  
import glob
from IPython.core.display import display, Image
from matplotlib import pyplot
from sklearn.utils import class_weight
from keras.callbacks import LearningRateScheduler, ModelCheckpoint
from keras.optimizers import SGD
from keras.preprocessing.image import ImageDataGenerator
from keras import backend as K
K.set_image_dim_ordering('th')


from helpers.preprocessImage import *
from helpers.imageDetails import randomImageDetails
from helpers.classDetails import datasetDetails
from helpers.classLabels import loadClassLabelsMapAndList
from helpers.simpleCNN import get_simple_cnn_model
from helpers.lenetModel import get_lenet_model
from helpers.modifiedLenet import get_modified_lenet_model
from helpers.modelPerformance import show_models_performance
from helpers.testImages import detect_image_type_lenet_model
from helpers.stratify import get_stratified_dataset


#Incoming image size after preprocessing - we resize the image to be of size 32x32
IMAGE_SIZE = 32

#Total class labels for the dataset
TOTAL_CLASSES = 43
Using TensorFlow backend.

PREPROCESSING DATA

Read and preprocess all the images from folder ./GTSRB/Final_Training/Images.

These images are from the training dataset. The readPreprocessedTrainTrafficSigns method reads the images from the folders and their corresponding labels from the csv files. It then processes the images and stores in the images in an array.

Lastly, the method returns preprocessed train images array and an array of corresponsing labels for each image.

We then convert the images array to a numpy array. We also perform one hot encoding for the labels array and store it in labels.

In [2]:
images_arr, labels_arr = readPreprocessedTrainTrafficSigns('./GTSRB/Final_Training/Images', IMAGE_SIZE)
images = np.array(images_arr, dtype='float32')
labels = np_utils.to_categorical(np.array(labels_arr), 43)

Read and preprocess all the images from folder ./GTSRB/Final_Test/Images.

These images are from the testing dataset. The readPreprocessedTestTrafficSigns method reads the images from the folders and their corresponding labels from the csv files. It then processes the images and stores in the images in an array.

Lastly, the method returns preprocessed test images array and an array of corresponsing labels for each image.

We then convert the test images array and test labels array to a numpy array.

In [3]:
test_images_arr, test_labels_arr = readPreprocessedTestTrafficSigns('./GTSRB/Final_Test/Images', IMAGE_SIZE)
test_images = np.array(test_images_arr, dtype='float32')
test_labels = np.array(test_labels_arr)

DISPLAYING IMAGE DETAILS

TRAIN DATASET

Let's print the details of any image from the training dataset. We have a method called randomImageDetails. This method takes some random number within the length of the images array (index for the train images array) and prints out details of the image.

For example - If random_number = 5, we print out the details of the 5th image from the train images array and labels array.

In [4]:
"""
Picking a random TRAIN image and printing its details -
1. Printing the image array
2. Printing the image shape
3. Printing the total rows and columns in the image
4. Printing the label for the image

We also plot the processed image
"""
random_number = randint(0, len(images))
randomImageDetails(random_number, images, labels_arr)
The image 8272 is: 
[[[ 0.21113184  0.21029459  0.21233572 ...,  0.74138349  0.70878738
    0.79364097]
  [ 0.21766734  0.19957784  0.17807133 ...,  0.68432212  0.70918226
    0.66648346]
  [ 0.19566247  0.21924214  0.18034428 ...,  0.66862005  0.7921378
    0.72526491]
  ..., 
  [ 0.14166875  0.03965906  0.01205688 ...,  0.73304504  0.82815754
    0.85547501]
  [ 0.06262078  0.036604    0.00473145 ...,  0.37369236  0.54923964
    0.5646854 ]
  [ 0.00976379  0.03285702  0.00174055 ...,  0.24750243  0.30574062
    0.3145597 ]]

 [[ 0.21632718  0.20766592  0.20730314 ...,  0.76573873  0.67369944
    0.74466276]
  [ 0.22029655  0.19957784  0.17584676 ...,  0.69382328  0.68428481
    0.67633337]
  [ 0.19566247  0.21402122  0.16678594 ...,  0.67509431  0.76778984
    0.67375737]
  ..., 
  [ 0.14166875  0.03965906  0.01205688 ...,  0.74381125  0.78851128
    0.86838514]
  [ 0.06417878  0.036604    0.00464844 ...,  0.3965621   0.54232568
    0.56131947]
  [ 0.00976379  0.03342788  0.00170832 ...,  0.24750243  0.30574062
    0.3198286 ]]

 [[ 0.24197835  0.25235352  0.23001128 ...,  0.83559448  0.69955659
    0.71310157]
  [ 0.27537718  0.24077474  0.19645183 ...,  0.75691569  0.78350914
    0.74796999]
  [ 0.22857259  0.25578856  0.18910319 ...,  0.67561436  0.84085816
    0.84094238]
  ..., 
  [ 0.18571167  0.04762695  0.01369629 ...,  0.6931529   0.76793301
    0.89199096]
  [ 0.07991944  0.04207275  0.00498047 ...,  0.41755411  0.57375705
    0.63421267]
  [ 0.01079102  0.04044556  0.00180501 ...,  0.28699464  0.3518917
    0.38066366]]]

Image 8272 shape is: 
(3, 32, 32)

In image 8272: Total number of rows: 32 Total number of columns: 32

The label for image 8272 is: 5
Plotting the image..

TEST DATASET

Let's print the details of any image from the testing dataset. This is similar to the code cell above except this is for the testing dataset.

For example - If random_number = 5, we print out the details of the 5th image from the test images array and labels array.

In [5]:
"""
Picking a random TEST image and printing its details -
1. Printing the image array
2. Printing the image shape
3. Printing the total rows and columns in the image
4. Printing the label for the image

We also plot the processed image
"""

random_number = randint(0, len(test_images))
randomImageDetails(random_number, test_images, test_labels_arr)
The image 5983 is: 
[[[ 0.01017033  0.01278553  0.04489349 ...,  0.7142787   0.74254823
    0.79493797]
  [ 0.05242205  0.02625098  0.0142867  ...,  0.76394081  0.76810193
    0.7980321 ]
  [ 0.03818803  0.05072714  0.03832281 ...,  0.67467707  0.72484541
    0.77888793]
  ..., 
  [ 0.51136333  0.16362011  0.17644881 ...,  0.32966179  0.19606304
    0.12287743]
  [ 0.34456098  0.14214157  0.17536342 ...,  0.23033522  0.2718485
    0.19251372]
  [ 0.31549436  0.10376801  0.15285155 ...,  0.23909692  0.27865496
    0.2408952 ]]

 [[ 0.00910104  0.01185582  0.03850999 ...,  0.77454603  0.7948814
    0.86241096]
  [ 0.04615184  0.02450099  0.01417802 ...,  0.7843523   0.7872448
    0.84248596]
  [ 0.03502806  0.04601847  0.0361104  ...,  0.69560415  0.74905968
    0.81493843]
  ..., 
  [ 0.43629685  0.14701109  0.1524263  ...,  0.31668037  0.20038211
    0.12503855]
  [ 0.31511676  0.13063255  0.15377177 ...,  0.22425021  0.28454405
    0.20434858]
  [ 0.28411037  0.0982257   0.14187191 ...,  0.22659722  0.27867362
    0.2463319 ]]

 [[ 0.00886363  0.01058268  0.03516414 ...,  0.83892131  0.84629506
    0.92177385]
  [ 0.04674105  0.02195963  0.0127188  ...,  0.835778    0.7946226
    0.87074476]
  [ 0.03561575  0.04227806  0.03023469 ...,  0.73448241  0.67728305
    0.79442793]
  ..., 
  [ 0.39219484  0.11769862  0.11988524 ...,  0.32555455  0.18886
    0.11437164]
  [ 0.27208665  0.10405611  0.11263531 ...,  0.2135849   0.26245132
    0.18061593]
  [ 0.23847023  0.07362176  0.10803255 ...,  0.19843748  0.23546191
    0.2013956 ]]]

Image 5983 shape is: 
(3, 32, 32)

In image 5983: Total number of rows: 32 Total number of columns: 32

The label for image 5983 is: 25
Plotting the image..

ANALYSING THE DATASET

Let's analyse the dataset. We will first be mapping some meaningful traffic sign names for each class ID.

We use the following link to get some meaningful names for each kind of traffic sign image - http://www.gettingaroundgermany.info/zeichen.shtml

After mapping this, we display a graph of the frequency of images for each class ID i.e. we find how many images do we have for each class label.

Since, the number of images for each class are not the same, we have an imbalanced dataset. The graph displays an imbalanced dataset.

We also print out the frequency of images for each class, the class with the highest frequency and the class with the lowest frequency.

In [6]:
#Loading the mapping between class IDs and labels
class_labels_list = loadClassLabelsMapAndList('./GTSRB')

#Visualizing the class labels frequency. 
#Also, prints the frequency for each class ID.
datasetDetails(labels_arr, TOTAL_CLASSES, class_labels_list)
Class 2 has the highest frequency and the frequency is 2250
Class 0 has the lowest frequency and the frequency is 210
******* Printing Class ID, Class Label and Frequency Mapping *******
Class ID 0 - ”Speed Limit 20” - Frequency 210
Class ID 1 - ”Speed Limit 30” - Frequency 2220
Class ID 2 - ”Speed Limit 50” - Frequency 2250
Class ID 3 - ”Speed Limit 60” - Frequency 1410
Class ID 4 - ”Speed Limit 70” - Frequency 1980
Class ID 5 - ”Speed Limit 80” - Frequency 1860
Class ID 6 - ”End of speed limit 80” - Frequency 420
Class ID 7 - ”Speed Limit 100” - Frequency 1440
Class ID 8 - ”Speed Limit 120” - Frequency 1410
Class ID 9 - ”No Passing” - Frequency 1470
Class ID 10 - ”No Passing for vehicles over 3.5t” - Frequency 2010
Class ID 11 - ”Priority” - Frequency 1320
Class ID 12 - ”Priority Road” - Frequency 2100
Class ID 13 - ”Yield” - Frequency 2160
Class ID 14 - ”Stop” - Frequency 780
Class ID 15 - ”Road Closed” - Frequency 630
Class ID 16 - ”Vehicles over 3.5t prohibited” - Frequency 420
Class ID 17 - ”Do Not Enter” - Frequency 1110
Class ID 18 - ”General Danger” - Frequency 1200
Class ID 19 - ”Curve (left)” - Frequency 210
Class ID 20 - ”Curve (right)” - Frequency 360
Class ID 21 - ”Double Curve. First curve is to the left” - Frequency 330
Class ID 22 - ”Rough Road” - Frequency 390
Class ID 23 - ”Slippery when wet or dirty” - Frequency 510
Class ID 24 - ”Road narrows (right side)” - Frequency 270
Class ID 25 - ”Road Work” - Frequency 1500
Class ID 26 - ”Traffic signals ahead” - Frequency 600
Class ID 27 - ”Pedestrians” - Frequency 240
Class ID 28 - ”Watch for Children” - Frequency 540
Class ID 29 - ”Bicycle Crossing” - Frequency 270
Class ID 30 - ”Beware of ice/snow” - Frequency 450
Class ID 31 - ”Wild animal crossing” - Frequency 780
Class ID 32 - ”End of all restrictions” - Frequency 240
Class ID 33 - ”Mandatory Direction of travel. All traffic must turn right” - Frequency 689
Class ID 34 - ”Mandatory Direction of travel. All traffic must turn left” - Frequency 420
Class ID 35 - ”Mandatory Direction of travel. All traffic must continue straight ahead” - Frequency 1200
Class ID 36 - ”Mandatory Direction of travel. All traffic must continue straight ahead or turn right” - Frequency 390
Class ID 37 - ”Mandatory Direction of travel. All traffic must continue straight ahead or turn left” - Frequency 210
Class ID 38 - ”Pass by on right” - Frequency 2070
Class ID 39 - ”Pass by on left” - Frequency 300
Class ID 40 - ”Roundabout” - Frequency 360
Class ID 41 - ”End of No Passing Zone” - Frequency 240
Class ID 42 - ”End of No Passing Zone for vehicles over 3.5t” - Frequency 240

SIMPLE CNN MODEL

Let's first try training a simple CNN model using our preprocessed images.

We use the test_train_split method to split the dataset into train and validation set. 80% images are used for training and 20% are used for validation. We will not be using data augmentation for this model.

We increased the number of filters with every convolutional layer which in turn increases the depth of every convolutional layer.

We alternated every convolutional layer with a max pooling layer. Pooling layers are used to reduce the spatial size of the representation to reduce the amount of parameters and computation.

I also added a global average pooling layer. This will reduce the 3D array to a vector without losing any information.

The last layer is the output layer (dense layer) and it has 43 nodes (as there are 43 types of traffic signs).

We will also print out the model's performance using our method called show_models_performance.

In [7]:
#Simple CNN model without data augmentation and without stratified data split

simple_model = get_simple_cnn_model(IMAGE_SIZE)
simple_model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])
checkpointer = ModelCheckpoint(filepath='best_simple_cnn.hdf5', verbose=1, save_best_only=True)
simple_model_history = simple_model.fit(images, labels, validation_split=0.2, shuffle=True, epochs=15, batch_size=32, callbacks=[checkpointer], verbose=1)
simple_cnn_y_pred = simple_model.predict_classes(test_images)
show_models_performance(test_labels, simple_cnn_y_pred, TOTAL_CLASSES)
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 64, 32, 32)        832       
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 64, 16, 16)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 128, 16, 16)       32896     
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 128, 8, 8)         0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 256, 8, 8)         131328    
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 256, 4, 4)         0         
_________________________________________________________________
global_average_pooling2d_1 ( (None, 256)               0         
_________________________________________________________________
dense_1 (Dense)              (None, 43)                11051     
=================================================================
Total params: 176,107.0
Trainable params: 176,107.0
Non-trainable params: 0.0
_________________________________________________________________
Train on 31367 samples, validate on 7842 samples
Epoch 1/15
31296/31367 [============================>.] - ETA: 0s - loss: 2.6976 - acc: 0.2213Epoch 00000: val_loss improved from inf to 16.11791, saving model to best_simple_cnn.hdf5
31367/31367 [==============================] - 24s - loss: 2.6964 - acc: 0.2216 - val_loss: 16.1179 - val_acc: 0.0000e+00
Epoch 2/15
31328/31367 [============================>.] - ETA: 0s - loss: 1.5789 - acc: 0.5165Epoch 00001: val_loss did not improve
31367/31367 [==============================] - 23s - loss: 1.5785 - acc: 0.5167 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 3/15
31328/31367 [============================>.] - ETA: 0s - loss: 0.9842 - acc: 0.6926Epoch 00002: val_loss did not improve
31367/31367 [==============================] - 23s - loss: 0.9840 - acc: 0.6927 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 4/15
31328/31367 [============================>.] - ETA: 0s - loss: 0.6343 - acc: 0.8019Epoch 00003: val_loss did not improve
31367/31367 [==============================] - 23s - loss: 0.6341 - acc: 0.8020 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 5/15
31328/31367 [============================>.] - ETA: 0s - loss: 0.4247 - acc: 0.8681Epoch 00004: val_loss did not improve
31367/31367 [==============================] - 23s - loss: 0.4245 - acc: 0.8681 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 6/15
31328/31367 [============================>.] - ETA: 0s - loss: 0.3027 - acc: 0.9064Epoch 00005: val_loss did not improve
31367/31367 [==============================] - 23s - loss: 0.3027 - acc: 0.9065 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 7/15
31328/31367 [============================>.] - ETA: 0s - loss: 0.2248 - acc: 0.9309- ETA: 1s - loss: Epoch 00006: val_loss did not improve
31367/31367 [==============================] - 23s - loss: 0.2247 - acc: 0.9309 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 8/15
31328/31367 [============================>.] - ETA: 0s - loss: 0.1704 - acc: 0.9488Epoch 00007: val_loss did not improve
31367/31367 [==============================] - 23s - loss: 0.1703 - acc: 0.9489 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 9/15
31328/31367 [============================>.] - ETA: 0s - loss: 0.1342 - acc: 0.9601Epoch 00008: val_loss did not improve
31367/31367 [==============================] - 23s - loss: 0.1341 - acc: 0.9601 - val_loss: 16.1181 - val_acc: 0.0000e+00
Epoch 10/15
31328/31367 [============================>.] - ETA: 0s - loss: 0.1102 - acc: 0.9677Epoch 00009: val_loss improved from 16.11791 to 16.11781, saving model to best_simple_cnn.hdf5
31367/31367 [==============================] - 23s - loss: 0.1102 - acc: 0.9676 - val_loss: 16.1178 - val_acc: 0.0000e+00
Epoch 11/15
31328/31367 [============================>.] - ETA: 0s - loss: 0.0902 - acc: 0.9729Epoch 00010: val_loss improved from 16.11781 to 16.11773, saving model to best_simple_cnn.hdf5
31367/31367 [==============================] - 23s - loss: 0.0903 - acc: 0.9728 - val_loss: 16.1177 - val_acc: 0.0000e+00
Epoch 12/15
31328/31367 [============================>.] - ETA: 0s - loss: 0.0770 - acc: 0.9786Epoch 00011: val_loss did not improve
31367/31367 [==============================] - 23s - loss: 0.0770 - acc: 0.9786 - val_loss: 16.1178 - val_acc: 0.0000e+00
Epoch 13/15
31328/31367 [============================>.] - ETA: 0s - loss: 0.0645 - acc: 0.9821Epoch 00012: val_loss improved from 16.11773 to 16.11613, saving model to best_simple_cnn.hdf5
31367/31367 [==============================] - 23s - loss: 0.0644 - acc: 0.9821 - val_loss: 16.1161 - val_acc: 0.0000e+00
Epoch 14/15
31328/31367 [============================>.] - ETA: 0s - loss: 0.0550 - acc: 0.9845Epoch 00013: val_loss improved from 16.11613 to 16.11523, saving model to best_simple_cnn.hdf5
31367/31367 [==============================] - 23s - loss: 0.0549 - acc: 0.9845 - val_loss: 16.1152 - val_acc: 0.0000e+00
Epoch 15/15
31328/31367 [============================>.] - ETA: 0s - loss: 0.0482 - acc: 0.9864Epoch 00014: val_loss did not improve
31367/31367 [==============================] - 23s - loss: 0.0483 - acc: 0.9864 - val_loss: 16.1163 - val_acc: 0.0000e+00
12630/12630 [==============================] - 2s     

Lenet Model accuracy for the test dataset: 0.6697545526524149

When Average is micro - Calculate metrics globally by counting the total true positives, false negatives and false positives.
precision: 0.6697545526524149
recall: 0.6697545526524149
fscore: 0.6697545526524149
support: None

When Average is macro - Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account.
precision: 0.5321627471745951
recall: 0.5474291277198254
fscore: 0.5175855554797817
support: None

When Average is weighted - Calculate metrics for each label, and find their average weighted by support (the number of true instances for each label). This alters ‘macro’ to account for label imbalance; it can result in an F-score that is not between precision and recall.
precision: 0.6523313976657656
recall: 0.6697545526524149
fscore: 0.6280193204371762
support: None

Classification Report
             precision    recall  f1-score   support

          0    0.68571   0.40000   0.50526        60
          1    0.96370   0.81111   0.88084       720
          2    0.93695   0.71333   0.80999       750
          3    1.00000   0.22222   0.36364       450
          4    0.99290   0.84697   0.91415       660
          5    0.32589   0.97302   0.48825       630
          6    0.31111   0.56000   0.40000       150
          7    0.71306   0.92222   0.80426       450
          8    0.69048   0.77333   0.72956       450
          9    0.98226   0.92292   0.95166       480
         10    0.99822   0.85152   0.91905       660
         11    0.91453   0.76429   0.83268       420
         12    0.30157   1.00000   0.46340       690
         13    0.97250   0.93333   0.95252       720
         14    0.99248   0.97778   0.98507       270
         15    0.97549   0.94762   0.96135       210
         16    0.97902   0.93333   0.95563       150
         17    1.00000   0.86389   0.92697       360
         18    0.89751   0.83077   0.86285       390
         19    1.00000   0.88333   0.93805        60
         20    0.40571   0.78889   0.53585        90
         21    0.54412   0.41111   0.46835        90
         22    0.88000   0.73333   0.80000       120
         23    0.42160   0.80667   0.55378       150
         24    0.94444   0.94444   0.94444        90
         25    0.79823   0.93958   0.86316       480
         26    0.81340   0.94444   0.87404       180
         27    0.94737   0.90000   0.92308        60
         28    0.49474   0.94000   0.64828       150
         29    0.00000   0.00000   0.00000        90
         30    0.00000   0.00000   0.00000       150
         31    0.00000   0.00000   0.00000       270
         32    0.00000   0.00000   0.00000        60
         33    0.00000   0.00000   0.00000       210
         34    0.00000   0.00000   0.00000       120
         35    0.00000   0.00000   0.00000       390
         36    0.00000   0.00000   0.00000       120
         37    0.00000   0.00000   0.00000        60
         38    0.00000   0.00000   0.00000       690
         39    0.00000   0.00000   0.00000        90
         40    0.00000   0.00000   0.00000        90
         41    0.00000   0.00000   0.00000        60
         42    0.00000   0.00000   0.00000        90

avg / total    0.65233   0.66975   0.62802     12630


Confusion Matrix
[[ 24   4   0 ...,   0   0   0]
 [  0 584   4 ...,   0   0   0]
 [  0  10 535 ...,   0   0   0]
 ..., 
 [  0   0   0 ...,   0   0   0]
 [  0   0   0 ...,   0   0   0]
 [  0   0   0 ...,   0   0   0]]
/usr/local/lib/python3.5/dist-packages/sklearn/metrics/classification.py:1113: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.
  'precision', 'predicted', average, warn_for)

We can see the performance of our model on the validation set. We provide only 15 epochs for training. We see that the validation loss does not improve and hence, we stopped the training of the model.

We have printed the model summary.

Since, our dataset is imbalanced, accuracy is not the best metric to use for finding out how our model is performing. We will see the performance of our model using precision, recall and F-beta score. We will use the weighted average version of the method precision_recall_fscore_support.

We print out the classification report for the model.

Confusion matrix is calculated using the confusion_matrix method of sklearn.metrics. We use matplotlib and seaborn modules to plot the confusion matrix.

PLOTTING LOSS VS EPOCHS

We have displayed a plot of loss on training and validation datasets over training epochs for simple CNN model.

In [8]:
#Plot Loss values for training and validation
pyplot.plot(simple_model_history.history['loss'])
pyplot.plot(simple_model_history.history['val_loss'])
pyplot.title('Model loss')
pyplot.ylabel('Loss')
pyplot.xlabel('Epochs')
pyplot.legend(['Train', 'Test'], loc='upper right')
pyplot.show()

SIMPLE CNN MODEL PERFORMANCE ON IMAGES FROM THE WEB

Now, let's see how our simple CNN models performs on randomly selected images from the web.

Our detect_image_type_lenet_model prints out the prediction for the input image.

In [9]:
img_list = sorted(glob.glob("./test_traffic_sign_images/*"))
    
for img_path in img_list:
    display_img = Image(img_path, width=175, height=175)
    display(display_img)
    detect_image_type_lenet_model(simple_model, img_path, class_labels_list, IMAGE_SIZE)
The predicted traffic sign is ”Speed Limit 30”
The predicted traffic sign is ”No Passing”
The predicted traffic sign is ”Watch for Children”
The predicted traffic sign is ”Speed Limit 50”
The predicted traffic sign is ”Priority Road”
The predicted traffic sign is ”Do Not Enter”
The predicted traffic sign is ”Speed Limit 120”
The predicted traffic sign is ”Watch for Children”
The predicted traffic sign is ”Road Work”
The predicted traffic sign is ”Speed Limit 120”
The predicted traffic sign is ”Road Work”
The predicted traffic sign is ”Priority Road”
The predicted traffic sign is ”Priority Road”
The predicted traffic sign is ”Priority Road”
The predicted traffic sign is ”Yield”
The predicted traffic sign is ”Yield”

You can see that the simple CNN model does not perform so well on the images picked up from the web. It correctly classifies only 8 out of 16 images that we provide. It correctly classifies 50% of the images we provide.

The model is clearly underfitting. So, we try to improve it ahead by different techniques.

LENET-5 MODELS

We will now be trying the Lenet-5 model and the modified Lenet-5 model.

COMMON FUNCTION CALLS FOR THE MODELS

The function calls below are some common function calls made for both the models that we will be training.

We have a method called get_stratified_dataset which in turn calls the StratifiedShuffleSplit of sklearn. Since, our dataset is imbalanced, this method will help us create a train validation dataset which is equally balanced in terms of the classes.

In addition, we will also use class weights when we fit the model. When our model comes across a training image from the class which has less samples, our model will pay more attention when calculating the loss. We calculate the class weights by using the compute_class_weights method of the sklearn.utils module.

In [13]:
X_train, Y_train, X_val, Y_val = get_stratified_dataset(images, labels)

class_weights = class_weight.compute_class_weight('balanced', np.unique(labels_arr), labels_arr)
print("Class weights are: {}".format(class_weights))
Class weights are: [ 4.34208195  0.41073748  0.40526098  0.64669306  0.46052384  0.49023506
  2.17104097  0.63322028  0.64669306  0.62029742  0.45365035  0.69078576
  0.43420819  0.42214686  1.16902206  1.44736065  2.17104097  0.82147496
  0.75986434  4.34208195  2.53288114  2.76314306  2.33804413  1.7879161
  3.37717485  0.60789147  1.51972868  3.79932171  1.68858742  3.37717485
  2.02630491  1.16902206  3.79932171  1.3234212   2.17104097  0.75986434
  2.33804413  4.34208195  0.44050107  3.03945736  2.53288114  3.79932171
  3.79932171]

LENET-5 MODEL - COMPILE AND FIT THE MODEL

We are now trying out the Lenet-5 model for the dataset.

We will be using the ImageDataGenerator method to do data augmentation.

We print out the summary of the Lenet-5 model.

We will be training it for only 25 epochs, as we have noticed that the validation loss does not improve after that. So, we stop the training after 25 epochs else the model will start overfitting.

You can notice that we have used the train set and validation set which we created in the above code cell using StratifiedShuffleSplit.

We will be using the SGD optimizer as it has better generalization than adaptive optimizers. We are using a decreasing learning rate with SGD. The learning rate decreases with every epoch. This helps us take smaller steps towards the solution. As we go closer to the solution, we want to take smaller steps towards it so that we do not jump over the solution.

In [14]:
lr = 0.01
lenet_sgd_optimizer = SGD(lr=lr, decay=1e-6, momentum=0.9, nesterov=True)

def learning_rate_scheduler(epoch):
    return lr*(0.1**int(epoch/10))

datagen_lenet = ImageDataGenerator(rotation_range=17,
                                            width_shift_range=0.1,
                                            height_shift_range=0.1,
                                            shear_range=0.3,
                                            zoom_range=0.15,
                                            horizontal_flip=False,
                                            fill_mode='nearest')

datagen_lenet.fit(X_train)
datagen_lenet_model = get_lenet_model(IMAGE_SIZE, TOTAL_CLASSES)
datagen_lenet_model.summary()

datagen_lenet_model.compile(optimizer=lenet_sgd_optimizer, 
                            loss='categorical_crossentropy', 
                            metrics=['accuracy'])

datagen_lenet_checkpointer = ModelCheckpoint(filepath='best_lenet_datagen.hdf5', 
                                             verbose=1,
                                             save_best_only=True)

datagen_lenet_model_history = datagen_lenet_model.fit_generator(datagen_lenet.flow(X_train, Y_train, batch_size=32),
                            class_weight=class_weights,
                            steps_per_epoch=X_train.shape[0] / 8,
                            epochs=25,
                            validation_data=(X_val, Y_val),
                            callbacks=[LearningRateScheduler(learning_rate_scheduler), datagen_lenet_checkpointer])
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_6 (Conv2D)            (None, 6, 28, 28)         456       
_________________________________________________________________
max_pooling2d_6 (MaxPooling2 (None, 6, 14, 14)         0         
_________________________________________________________________
conv2d_7 (Conv2D)            (None, 16, 10, 10)        2416      
_________________________________________________________________
max_pooling2d_7 (MaxPooling2 (None, 16, 5, 5)          0         
_________________________________________________________________
flatten_2 (Flatten)          (None, 400)               0         
_________________________________________________________________
dense_5 (Dense)              (None, 120)               48120     
_________________________________________________________________
dense_6 (Dense)              (None, 84)                10164     
_________________________________________________________________
dense_7 (Dense)              (None, 43)                3655      
=================================================================
Total params: 64,811.0
Trainable params: 64,811.0
Non-trainable params: 0.0
_________________________________________________________________
Epoch 1/25
3918/3920 [============================>.] - ETA: 0s - loss: 1.0779 - acc: 0.6667Epoch 00000: val_loss improved from inf to 0.24015, saving model to best_lenet_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 1.0774 - acc: 0.6668 - val_loss: 0.2402 - val_acc: 0.9211
Epoch 2/25
3920/3920 [============================>.] - ETA: 0s - loss: 0.5061 - acc: 0.8390Epoch 00001: val_loss improved from 0.24015 to 0.20222, saving model to best_lenet_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.5061 - acc: 0.8390 - val_loss: 0.2022 - val_acc: 0.9371
Epoch 3/25
3920/3920 [============================>.] - ETA: 0s - loss: 0.4216 - acc: 0.8686Epoch 00002: val_loss improved from 0.20222 to 0.18198, saving model to best_lenet_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.4215 - acc: 0.8686 - val_loss: 0.1820 - val_acc: 0.9412
Epoch 4/25
3919/3920 [============================>.] - ETA: 0s - loss: 0.3654 - acc: 0.8861Epoch 00003: val_loss improved from 0.18198 to 0.13678, saving model to best_lenet_datagen.hdf5
3921/3920 [==============================] - 48s - loss: 0.3653 - acc: 0.8861 - val_loss: 0.1368 - val_acc: 0.9558
Epoch 5/25
3917/3920 [============================>.] - ETA: 0s - loss: 0.3432 - acc: 0.8949Epoch 00004: val_loss improved from 0.13678 to 0.11103, saving model to best_lenet_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.3431 - acc: 0.8949 - val_loss: 0.1110 - val_acc: 0.9635
Epoch 6/25
3919/3920 [============================>.] - ETA: 0s - loss: 0.3209 - acc: 0.9022Epoch 00005: val_loss did not improve
3921/3920 [==============================] - 49s - loss: 0.3209 - acc: 0.9022 - val_loss: 0.1358 - val_acc: 0.9570
Epoch 7/25
3918/3920 [============================>.] - ETA: 0s - loss: 0.3121 - acc: 0.9051Epoch 00006: val_loss improved from 0.11103 to 0.10092, saving model to best_lenet_datagen.hdf5
3921/3920 [==============================] - 48s - loss: 0.3122 - acc: 0.9051 - val_loss: 0.1009 - val_acc: 0.9698
Epoch 8/25
3919/3920 [============================>.] - ETA: 0s - loss: 0.2999 - acc: 0.9097Epoch 00007: val_loss improved from 0.10092 to 0.09072, saving model to best_lenet_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.2999 - acc: 0.9096 - val_loss: 0.0907 - val_acc: 0.9712
Epoch 9/25
3920/3920 [============================>.] - ETA: 0s - loss: 0.2923 - acc: 0.9132Epoch 00008: val_loss improved from 0.09072 to 0.08299, saving model to best_lenet_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.2924 - acc: 0.9132 - val_loss: 0.0830 - val_acc: 0.9745
Epoch 10/25
3918/3920 [============================>.] - ETA: 0s - loss: 0.2813 - acc: 0.9162Epoch 00009: val_loss did not improve
3921/3920 [==============================] - 49s - loss: 0.2812 - acc: 0.9162 - val_loss: 0.0897 - val_acc: 0.9727
Epoch 11/25
3920/3920 [============================>.] - ETA: 0s - loss: 0.1636 - acc: 0.9495Epoch 00010: val_loss improved from 0.08299 to 0.03966, saving model to best_lenet_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.1636 - acc: 0.9495 - val_loss: 0.0397 - val_acc: 0.9876
Epoch 12/25
3917/3920 [============================>.] - ETA: 0s - loss: 0.1394 - acc: 0.9564Epoch 00011: val_loss improved from 0.03966 to 0.03512, saving model to best_lenet_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.1395 - acc: 0.9564 - val_loss: 0.0351 - val_acc: 0.9883
Epoch 13/25
3917/3920 [============================>.] - ETA: 0s - loss: 0.1345 - acc: 0.9575Epoch 00012: val_loss improved from 0.03512 to 0.03297, saving model to best_lenet_datagen.hdf5
3921/3920 [==============================] - 48s - loss: 0.1345 - acc: 0.9575 - val_loss: 0.0330 - val_acc: 0.9889
Epoch 14/25
3916/3920 [============================>.] - ETA: 0s - loss: 0.1255 - acc: 0.9605Epoch 00013: val_loss improved from 0.03297 to 0.03118, saving model to best_lenet_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.1255 - acc: 0.9605 - val_loss: 0.0312 - val_acc: 0.9898
Epoch 15/25
3920/3920 [============================>.] - ETA: 0s - loss: 0.1243 - acc: 0.9609Epoch 00014: val_loss did not improve
3921/3920 [==============================] - 49s - loss: 0.1243 - acc: 0.9609 - val_loss: 0.0318 - val_acc: 0.9904
Epoch 16/25
3918/3920 [============================>.] - ETA: 0s - loss: 0.1198 - acc: 0.9623Epoch 00015: val_loss improved from 0.03118 to 0.02903, saving model to best_lenet_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.1198 - acc: 0.9623 - val_loss: 0.0290 - val_acc: 0.9898
Epoch 17/25
3916/3920 [============================>.] - ETA: 0s - loss: 0.1169 - acc: 0.9629Epoch 00016: val_loss did not improve
3921/3920 [==============================] - 48s - loss: 0.1168 - acc: 0.9629 - val_loss: 0.0314 - val_acc: 0.9901
Epoch 18/25
3919/3920 [============================>.] - ETA: 0s - loss: 0.1139 - acc: 0.9637Epoch 00017: val_loss improved from 0.02903 to 0.02788, saving model to best_lenet_datagen.hdf5
3921/3920 [==============================] - 48s - loss: 0.1139 - acc: 0.9637 - val_loss: 0.0279 - val_acc: 0.9912
Epoch 19/25
3917/3920 [============================>.] - ETA: 0s - loss: 0.1120 - acc: 0.9641Epoch 00018: val_loss did not improve
3921/3920 [==============================] - 49s - loss: 0.1120 - acc: 0.9641 - val_loss: 0.0283 - val_acc: 0.9912
Epoch 20/25
3917/3920 [============================>.] - ETA: 0s - loss: 0.1110 - acc: 0.9651Epoch 00019: val_loss did not improve
3921/3920 [==============================] - 48s - loss: 0.1110 - acc: 0.9650 - val_loss: 0.0294 - val_acc: 0.9907
Epoch 21/25
3918/3920 [============================>.] - ETA: 0s - loss: 0.1034 - acc: 0.9669Epoch 00020: val_loss improved from 0.02788 to 0.02630, saving model to best_lenet_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.1034 - acc: 0.9669 - val_loss: 0.0263 - val_acc: 0.9915
Epoch 22/25
3916/3920 [============================>.] - ETA: 0s - loss: 0.1025 - acc: 0.9673Epoch 00021: val_loss improved from 0.02630 to 0.02609, saving model to best_lenet_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.1025 - acc: 0.9673 - val_loss: 0.0261 - val_acc: 0.9912
Epoch 23/25
3918/3920 [============================>.] - ETA: 0s - loss: 0.1014 - acc: 0.9675Epoch 00022: val_loss improved from 0.02609 to 0.02556, saving model to best_lenet_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.1014 - acc: 0.9675 - val_loss: 0.0256 - val_acc: 0.9909
Epoch 24/25
3916/3920 [============================>.] - ETA: 0s - loss: 0.1023 - acc: 0.9674Epoch 00023: val_loss did not improve
3921/3920 [==============================] - 48s - loss: 0.1022 - acc: 0.9674 - val_loss: 0.0258 - val_acc: 0.9917
Epoch 25/25
3918/3920 [============================>.] - ETA: 0s - loss: 0.0985 - acc: 0.9681Epoch 00024: val_loss did not improve
3921/3920 [==============================] - 49s - loss: 0.0985 - acc: 0.9681 - val_loss: 0.0265 - val_acc: 0.9907

PLOTTING LOSS VS EPOCHS

We have displayed a plot of loss on training and validation datasets over training epochs for the Lenet-5 model.

In [21]:
#Plot Loss values for training and validation
pyplot.plot(datagen_lenet_model_history.history['loss'])
pyplot.plot(datagen_lenet_model_history.history['val_loss'])
pyplot.title('Model loss')
pyplot.ylabel('Loss')
pyplot.xlabel('Epochs')
pyplot.legend(['Train', 'Test'], loc='upper right')
pyplot.show()

LENET-5 MODEL - TESTING THE MODEL

We will now be testing the Lenet-5 model on the test dataset.

We print the performance of the model using the show_models_performance function that we implemented.

Since, our dataset is imbalanced, accuracy is not the best metric to use for finding out how our model is performing. We will see the performance of our model using precision, recall and F-beta score. We will use the weighted average version of the method precision_recall_fscore_support.

We print out the classification report for the model.

Confusion matrix is calculated using the confusion_matrix method of sklearn.metrics. We use matplotlib and seaborn modules to plot the confusion matrix.

In [15]:
datagen_lenet_y_pred = datagen_lenet_model.predict_classes(test_images)
datagen_lenet_test_accuracy = np.sum(datagen_lenet_y_pred == test_labels)/np.size(datagen_lenet_y_pred)
show_models_performance(test_labels, datagen_lenet_y_pred, TOTAL_CLASSES)
12576/12630 [============================>.] - ETA: 0s
Lenet Model accuracy for the test dataset: 0.9426761678543151

When Average is micro - Calculate metrics globally by counting the total true positives, false negatives and false positives.
precision: 0.9426761678543151
recall: 0.9426761678543151
fscore: 0.9426761678543151
support: None

When Average is macro - Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account.
precision: 0.9118074714627287
recall: 0.9244574178770337
fscore: 0.9140043556522633
support: None

When Average is weighted - Calculate metrics for each label, and find their average weighted by support (the number of true instances for each label). This alters ‘macro’ to account for label imbalance; it can result in an F-score that is not between precision and recall.
precision: 0.9462952742817614
recall: 0.9426761678543151
fscore: 0.9426501460331946
support: None

Classification Report
             precision    recall  f1-score   support

          0    0.87931   0.85000   0.86441        60
          1    0.97524   0.98472   0.97996       720
          2    0.95065   0.97600   0.96316       750
          3    0.97596   0.90222   0.93764       450
          4    0.96600   0.94697   0.95639       660
          5    0.89953   0.90952   0.90450       630
          6    1.00000   0.62667   0.77049       150
          7    0.91258   0.95111   0.93145       450
          8    0.95078   0.94444   0.94760       450
          9    0.96378   0.99792   0.98055       480
         10    0.99531   0.96364   0.97921       660
         11    0.95059   0.96190   0.95621       420
         12    0.96562   0.97681   0.97118       690
         13    0.98628   0.99861   0.99241       720
         14    0.95522   0.94815   0.95167       270
         15    0.93981   0.96667   0.95305       210
         16    0.98684   1.00000   0.99338       150
         17    0.99688   0.88889   0.93979       360
         18    0.96845   0.78718   0.86846       390
         19    0.90909   1.00000   0.95238        60
         20    0.66667   0.86667   0.75362        90
         21    0.73418   0.64444   0.68639        90
         22    0.99091   0.90833   0.94783       120
         23    0.95918   0.94000   0.94949       150
         24    0.63971   0.96667   0.76991        90
         25    0.92570   0.96042   0.94274       480
         26    0.90306   0.98333   0.94149       180
         27    0.75385   0.81667   0.78400        60
         28    0.97279   0.95333   0.96296       150
         29    0.85577   0.98889   0.91753        90
         30    0.86290   0.71333   0.78102       150
         31    0.89416   0.90741   0.90074       270
         32    0.68966   1.00000   0.81633        60
         33    0.99526   1.00000   0.99762       210
         34    0.91603   1.00000   0.95618       120
         35    0.96373   0.95385   0.95876       390
         36    0.96721   0.98333   0.97521       120
         37    0.87879   0.96667   0.92063        60
         38    0.97813   0.97246   0.97529       690
         39    0.97701   0.94444   0.96045        90
         40    0.94318   0.92222   0.93258        90
         41    0.92857   0.86667   0.89655        60
         42    0.68333   0.91111   0.78095        90

avg / total    0.94630   0.94268   0.94265     12630


Confusion Matrix
[[ 51   7   0 ...,   0   0   0]
 [  0 709   0 ...,   0   0   0]
 [  0   4 732 ...,   0   0   0]
 ..., 
 [  0   0   0 ...,  83   0   0]
 [  0   0   0 ...,   0  52   1]
 [  0   3   0 ...,   2   3  82]]

LENET-5 MODEL PERFORMANCE FOR IMAGES FROM THE WEB

Now, let's see how our Lenet-5 model performs on randomly selected images from the web.

Our detect_image_type_lenet_model prints out the prediction for the input image.

In [16]:
img_list = sorted(glob.glob("./test_traffic_sign_images/*"))
    
for img_path in img_list:
    display_img = Image(img_path, width=175, height=175)
    display(display_img)
    detect_image_type_lenet_model(datagen_lenet_model, img_path, class_labels_list, IMAGE_SIZE)
The predicted traffic sign is ”Speed Limit 30”
The predicted traffic sign is ”No Passing”
The predicted traffic sign is ”Watch for Children”
The predicted traffic sign is ”No Passing”
The predicted traffic sign is ”Priority Road”
The predicted traffic sign is ”Do Not Enter”
The predicted traffic sign is ”Speed Limit 70”
The predicted traffic sign is ”Beware of ice/snow”
The predicted traffic sign is ”Priority”
The predicted traffic sign is ”Pass by on right”
The predicted traffic sign is ”Road Work”
The predicted traffic sign is ”Roundabout”
The predicted traffic sign is ”Roundabout”
The predicted traffic sign is ”Traffic signals ahead”
The predicted traffic sign is ”Yield”
The predicted traffic sign is ”Yield”

You can see that the Lenet-5 model performed better on the images picked up from the web. It correctly classifies 14 out of 16 images that we provide. It correctly classifies 87.50% of the images we provide.

LENET-5 MODIFIED MODEL - COMPILE AND FIT THE MODEL

We are now trying out the Lenet-5 modified model for the dataset.

Each filter in CNN is a feature map. Increasing the filters helps in capturing more detailed features of the dataset. We have modified it by increasing the number of filters in the first and the second convolutional layers. 1st convolutional layer - Number of filters increased from 6 to 12 2nd convolutional layer - Number of filters increased from 16 to 32

We also added a dropout layer after the flatten layer. Dropout is a regularization technique. It helps avoid overfitting by ignoring randomly selected neurons when the model is getting trained. This helps in building a network which generalizes better.

We will be using the ImageDataGenerator method to do data augmentation.

We print out the summary of the Lenet-5 model.

We will be training it for only 30 epochs, as we have noticed that the validation loss does not improve after that. So, we stop the training after 30 epochs else the model will start overfitting.

You can notice that we have used the train set and validation set which we created in the above code cell using StratifiedShuffleSplit.

We will be using the SGD optimizer as it has better generalization than adaptive optimizers. We are using a decreasing learning rate with SGD. The learning rate decreases with every epoch. This helps us take smaller steps towards the solution. As we go closer to the solution, we want to take smaller steps towards it so that we do not jump over the solution.

In [17]:
lr = 0.01
lenet_sgd_optimizer = SGD(lr=lr, decay=1e-6, momentum=0.9, nesterov=True)

def learning_rate_scheduler(epoch):
    return lr*(0.1**int(epoch/10))

datagen_modified_lenet = ImageDataGenerator(rotation_range=17,
                                            width_shift_range=0.1,
                                            height_shift_range=0.1,
                                            shear_range=0.3,
                                            zoom_range=0.15,
                                            horizontal_flip=False,
                                            fill_mode='nearest')

datagen_modified_lenet.fit(X_train)
datagen_modified_lenet_model = get_modified_lenet_model(IMAGE_SIZE, TOTAL_CLASSES)
datagen_modified_lenet_model.summary()
datagen_modified_lenet_model.compile(optimizer=lenet_sgd_optimizer, 
                                     loss='categorical_crossentropy', 
                                     metrics=['accuracy'])

datagen_modified_lenet_checkpointer = ModelCheckpoint(filepath='best_lenet_modified_datagen.hdf5',
                                                      verbose=1, 
                                                      save_best_only=True)

datagen_modified_lenet_model_history = datagen_modified_lenet_model.fit_generator(datagen_modified_lenet.flow(X_train, Y_train, batch_size=32),
                            class_weight=class_weights,
                            steps_per_epoch=X_train.shape[0] / 8,
                            epochs=30,
                            validation_data=(X_val, Y_val),
                            callbacks=[LearningRateScheduler(learning_rate_scheduler), datagen_modified_lenet_checkpointer])
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_8 (Conv2D)            (None, 12, 28, 28)        912       
_________________________________________________________________
max_pooling2d_8 (MaxPooling2 (None, 12, 14, 14)        0         
_________________________________________________________________
conv2d_9 (Conv2D)            (None, 32, 10, 10)        9632      
_________________________________________________________________
max_pooling2d_9 (MaxPooling2 (None, 32, 5, 5)          0         
_________________________________________________________________
flatten_3 (Flatten)          (None, 800)               0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 800)               0         
_________________________________________________________________
dense_8 (Dense)              (None, 120)               96120     
_________________________________________________________________
dense_9 (Dense)              (None, 84)                10164     
_________________________________________________________________
dense_10 (Dense)             (None, 43)                3655      
=================================================================
Total params: 120,483.0
Trainable params: 120,483.0
Non-trainable params: 0.0
_________________________________________________________________
Epoch 1/30
3918/3920 [============================>.] - ETA: 0s - loss: 1.5044 - acc: 0.5407Epoch 00000: val_loss improved from inf to 0.35851, saving model to best_lenet_modified_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 1.5040 - acc: 0.5408 - val_loss: 0.3585 - val_acc: 0.8791
Epoch 2/30
3918/3920 [============================>.] - ETA: 0s - loss: 0.7716 - acc: 0.7503Epoch 00001: val_loss improved from 0.35851 to 0.20726, saving model to best_lenet_modified_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.7717 - acc: 0.7503 - val_loss: 0.2073 - val_acc: 0.9323
Epoch 3/30
3919/3920 [============================>.] - ETA: 0s - loss: 0.6476 - acc: 0.7936Epoch 00002: val_loss improved from 0.20726 to 0.20313, saving model to best_lenet_modified_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.6476 - acc: 0.7936 - val_loss: 0.2031 - val_acc: 0.9346
Epoch 4/30
3918/3920 [============================>.] - ETA: 0s - loss: 0.5676 - acc: 0.8201Epoch 00003: val_loss improved from 0.20313 to 0.13934, saving model to best_lenet_modified_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.5677 - acc: 0.8201 - val_loss: 0.1393 - val_acc: 0.9547
Epoch 5/30
3918/3920 [============================>.] - ETA: 0s - loss: 0.5252 - acc: 0.8351Epoch 00004: val_loss improved from 0.13934 to 0.12792, saving model to best_lenet_modified_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.5252 - acc: 0.8351 - val_loss: 0.1279 - val_acc: 0.9573
Epoch 6/30
3918/3920 [============================>.] - ETA: 0s - loss: 0.4910 - acc: 0.8473Epoch 00005: val_loss improved from 0.12792 to 0.09038, saving model to best_lenet_modified_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.4909 - acc: 0.8473 - val_loss: 0.0904 - val_acc: 0.9707
Epoch 7/30
3918/3920 [============================>.] - ETA: 0s - loss: 0.4679 - acc: 0.8553Epoch 00006: val_loss did not improve
3921/3920 [==============================] - 49s - loss: 0.4678 - acc: 0.8554 - val_loss: 0.0917 - val_acc: 0.9744
Epoch 8/30
3920/3920 [============================>.] - ETA: 0s - loss: 0.4488 - acc: 0.8625Epoch 00007: val_loss did not improve
3921/3920 [==============================] - 49s - loss: 0.4488 - acc: 0.8625 - val_loss: 0.0958 - val_acc: 0.9730
Epoch 9/30
3916/3920 [============================>.] - ETA: 0s - loss: 0.4428 - acc: 0.8647Epoch 00008: val_loss improved from 0.09038 to 0.07625, saving model to best_lenet_modified_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.4427 - acc: 0.8647 - val_loss: 0.0762 - val_acc: 0.9770
Epoch 10/30
3917/3920 [============================>.] - ETA: 0s - loss: 0.4318 - acc: 0.8687Epoch 00009: val_loss did not improve
3921/3920 [==============================] - 49s - loss: 0.4319 - acc: 0.8686 - val_loss: 0.0789 - val_acc: 0.9744
Epoch 11/30
3919/3920 [============================>.] - ETA: 0s - loss: 0.2828 - acc: 0.9110Epoch 00010: val_loss improved from 0.07625 to 0.03705, saving model to best_lenet_modified_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.2829 - acc: 0.9110 - val_loss: 0.0370 - val_acc: 0.9898
Epoch 12/30
3916/3920 [============================>.] - ETA: 0s - loss: 0.2609 - acc: 0.9178Epoch 00011: val_loss improved from 0.03705 to 0.03306, saving model to best_lenet_modified_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.2608 - acc: 0.9178 - val_loss: 0.0331 - val_acc: 0.9915
Epoch 13/30
3919/3920 [============================>.] - ETA: 0s - loss: 0.2407 - acc: 0.9230Epoch 00012: val_loss improved from 0.03306 to 0.03238, saving model to best_lenet_modified_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.2407 - acc: 0.9230 - val_loss: 0.0324 - val_acc: 0.9913
Epoch 14/30
3916/3920 [============================>.] - ETA: 0s - loss: 0.2344 - acc: 0.9248Epoch 00013: val_loss improved from 0.03238 to 0.02996, saving model to best_lenet_modified_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.2344 - acc: 0.9248 - val_loss: 0.0300 - val_acc: 0.9925
Epoch 15/30
3920/3920 [============================>.] - ETA: 0s - loss: 0.2253 - acc: 0.9277Epoch 00014: val_loss did not improve
3921/3920 [==============================] - 49s - loss: 0.2253 - acc: 0.9277 - val_loss: 0.0307 - val_acc: 0.9927
Epoch 16/30
3918/3920 [============================>.] - ETA: 0s - loss: 0.2268 - acc: 0.9277Epoch 00015: val_loss improved from 0.02996 to 0.02895, saving model to best_lenet_modified_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.2269 - acc: 0.9277 - val_loss: 0.0290 - val_acc: 0.9923
Epoch 17/30
3918/3920 [============================>.] - ETA: 0s - loss: 0.2190 - acc: 0.9293Epoch 00016: val_loss did not improve
3921/3920 [==============================] - 49s - loss: 0.2189 - acc: 0.9293 - val_loss: 0.0290 - val_acc: 0.9927
Epoch 18/30
3916/3920 [============================>.] - ETA: 0s - loss: 0.2168 - acc: 0.9301Epoch 00017: val_loss improved from 0.02895 to 0.02745, saving model to best_lenet_modified_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.2167 - acc: 0.9301 - val_loss: 0.0275 - val_acc: 0.9938
Epoch 19/30
3916/3920 [============================>.] - ETA: 0s - loss: 0.2099 - acc: 0.9321Epoch 00018: val_loss improved from 0.02745 to 0.02493, saving model to best_lenet_modified_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.2099 - acc: 0.9321 - val_loss: 0.0249 - val_acc: 0.9939
Epoch 20/30
3919/3920 [============================>.] - ETA: 0s - loss: 0.2087 - acc: 0.9324Epoch 00019: val_loss did not improve
3921/3920 [==============================] - 49s - loss: 0.2086 - acc: 0.9324 - val_loss: 0.0266 - val_acc: 0.9929
Epoch 21/30
3918/3920 [============================>.] - ETA: 0s - loss: 0.1978 - acc: 0.9366Epoch 00020: val_loss improved from 0.02493 to 0.02461, saving model to best_lenet_modified_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.1979 - acc: 0.9366 - val_loss: 0.0246 - val_acc: 0.9935
Epoch 22/30
3919/3920 [============================>.] - ETA: 0s - loss: 0.1967 - acc: 0.9359Epoch 00021: val_loss improved from 0.02461 to 0.02438, saving model to best_lenet_modified_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.1967 - acc: 0.9359 - val_loss: 0.0244 - val_acc: 0.9931
Epoch 23/30
3917/3920 [============================>.] - ETA: 0s - loss: 0.1931 - acc: 0.9375Epoch 00022: val_loss improved from 0.02438 to 0.02373, saving model to best_lenet_modified_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.1932 - acc: 0.9375 - val_loss: 0.0237 - val_acc: 0.9934
Epoch 24/30
3918/3920 [============================>.] - ETA: 0s - loss: 0.1976 - acc: 0.9365Epoch 00023: val_loss did not improve
3921/3920 [==============================] - 49s - loss: 0.1976 - acc: 0.9365 - val_loss: 0.0242 - val_acc: 0.9932
Epoch 25/30
3920/3920 [============================>.] - ETA: 0s - loss: 0.1988 - acc: 0.9365Epoch 00024: val_loss did not improve
3921/3920 [==============================] - 49s - loss: 0.1988 - acc: 0.9365 - val_loss: 0.0238 - val_acc: 0.9938
Epoch 26/30
3919/3920 [============================>.] - ETA: 0s - loss: 0.1957 - acc: 0.9372Epoch 00025: val_loss did not improve
3921/3920 [==============================] - 49s - loss: 0.1957 - acc: 0.9372 - val_loss: 0.0237 - val_acc: 0.9935
Epoch 27/30
3919/3920 [============================>.] - ETA: 0s - loss: 0.1965 - acc: 0.9368Epoch 00026: val_loss did not improve
3921/3920 [==============================] - 49s - loss: 0.1965 - acc: 0.9368 - val_loss: 0.0239 - val_acc: 0.9935
Epoch 28/30
3920/3920 [============================>.] - ETA: 0s - loss: 0.1971 - acc: 0.9361Epoch 00027: val_loss improved from 0.02373 to 0.02342, saving model to best_lenet_modified_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.1971 - acc: 0.9361 - val_loss: 0.0234 - val_acc: 0.9943
Epoch 29/30
3919/3920 [============================>.] - ETA: 0s - loss: 0.1926 - acc: 0.9379Epoch 00028: val_loss improved from 0.02342 to 0.02338, saving model to best_lenet_modified_datagen.hdf5
3921/3920 [==============================] - 49s - loss: 0.1926 - acc: 0.9379 - val_loss: 0.0234 - val_acc: 0.9935
Epoch 30/30
3919/3920 [============================>.] - ETA: 0s - loss: 0.1925 - acc: 0.9380Epoch 00029: val_loss did not improve
3921/3920 [==============================] - 49s - loss: 0.1926 - acc: 0.9379 - val_loss: 0.0235 - val_acc: 0.9940

PLOTTING LOSS VS EPOCHS

We have displayed a plot of loss on training and validation datasets over training epochs for the Lenet-5 modified model.

In [20]:
#Plot Loss values for training and validation
pyplot.plot(datagen_modified_lenet_model_history.history['loss'])
pyplot.plot(datagen_modified_lenet_model_history.history['val_loss'])
pyplot.title('Model loss')
pyplot.ylabel('Loss')
pyplot.xlabel('Epochs')
pyplot.legend(['Train', 'Test'], loc='upper right')
pyplot.show()

LENET-5 MODIFIED MODEL - TESTING THE MODEL

We will now be testing the Lenet-5 modified model on the test dataset.

We print the performance of the model using the show_models_performance function that we implemented.

Since, our dataset is imbalanced, accuracy is not the best metric to use for finding out how our model is performing. We will see the performance of our model using precision, recall and F-beta score. We will use the weighted average version of the method precision_recall_fscore_support.

We print out the classification report for the model.

Confusion matrix is calculated using the confusion_matrix method of sklearn.metrics. We use matplotlib and seaborn modules to plot the confusion matrix.

In [18]:
datagen_modified_lenet_y_pred = datagen_modified_lenet_model.predict_classes(test_images)
datagen_modified_lenet_test_accuracy = np.sum(datagen_modified_lenet_y_pred == test_labels)/np.size(datagen_modified_lenet_y_pred)
show_models_performance(test_labels, datagen_modified_lenet_y_pred, TOTAL_CLASSES)
12630/12630 [==============================] - 1s     

Lenet Model accuracy for the test dataset: 0.9598574821852731

When Average is micro - Calculate metrics globally by counting the total true positives, false negatives and false positives.
precision: 0.9598574821852731
recall: 0.9598574821852731
fscore: 0.9598574821852731
support: None

When Average is macro - Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account.
precision: 0.9457347314373249
recall: 0.9400486255309306
fscore: 0.9388324482110373
support: None

When Average is weighted - Calculate metrics for each label, and find their average weighted by support (the number of true instances for each label). This alters ‘macro’ to account for label imbalance; it can result in an F-score that is not between precision and recall.
precision: 0.9623634471864845
recall: 0.9598574821852731
fscore: 0.9596077460914391
support: None

Classification Report
             precision    recall  f1-score   support

          0    1.00000   0.60000   0.75000        60
          1    0.96699   0.97639   0.97167       720
          2    0.98398   0.98267   0.98332       750
          3    0.99287   0.92889   0.95982       450
          4    0.96951   0.96364   0.96657       660
          5    0.91742   0.96984   0.94290       630
          6    0.97391   0.74667   0.84528       150
          7    0.96767   0.93111   0.94904       450
          8    0.92872   0.98444   0.95577       450
          9    0.98967   0.99792   0.99378       480
         10    0.99542   0.98788   0.99163       660
         11    0.97337   0.95714   0.96519       420
         12    0.98981   0.98551   0.98765       690
         13    0.98087   0.99722   0.98898       720
         14    0.90268   0.99630   0.94718       270
         15    0.95872   0.99524   0.97664       210
         16    0.99333   0.99333   0.99333       150
         17    1.00000   0.89167   0.94273       360
         18    0.97493   0.89744   0.93458       390
         19    1.00000   0.96667   0.98305        60
         20    0.66129   0.91111   0.76636        90
         21    0.98333   0.65556   0.78667        90
         22    0.99138   0.95833   0.97458       120
         23    0.81143   0.94667   0.87385       150
         24    0.97802   0.98889   0.98343        90
         25    0.94343   0.97292   0.95795       480
         26    0.95745   1.00000   0.97826       180
         27    0.96667   0.96667   0.96667        60
         28    0.98013   0.98667   0.98339       150
         29    0.75424   0.98889   0.85577        90
         30    0.83471   0.67333   0.74539       150
         31    0.88929   0.92222   0.90545       270
         32    0.82192   1.00000   0.90226        60
         33    0.99057   1.00000   0.99526       210
         34    0.95238   1.00000   0.97561       120
         35    0.99740   0.98205   0.98966       390
         36    1.00000   1.00000   1.00000       120
         37    1.00000   0.98333   0.99160        60
         38    0.99561   0.98551   0.99053       690
         39    1.00000   0.95556   0.97727        90
         40    0.97778   0.97778   0.97778        90
         41    0.94828   0.91667   0.93220        60
         42    0.77143   0.90000   0.83077        90

avg / total    0.96236   0.95986   0.95961     12630


Confusion Matrix
[[ 36  17   0 ...,   0   0   0]
 [  0 703   2 ...,   0   0   0]
 [  0   2 737 ...,   0   0   0]
 ..., 
 [  0   0   0 ...,  88   0   0]
 [  0   0   0 ...,   0  55   2]
 [  0   0   0 ...,   0   3  81]]

LENET-5 MODIFIED MODEL PERFORMANCE FOR IMAGES FROM THE WEB

Now, let's see how our Lenet-5 modified model performs on randomly selected images from the web.

Our detect_image_type_lenet_model prints out the prediction for the input image.

In [19]:
img_list = sorted(glob.glob("./test_traffic_sign_images/*"))
    
for img_path in img_list:
    display_img = Image(img_path, width=175, height=175)
    display(display_img)
    detect_image_type_lenet_model(datagen_modified_lenet_model, img_path, class_labels_list, IMAGE_SIZE)
The predicted traffic sign is ”Speed Limit 30”
The predicted traffic sign is ”No Passing”
The predicted traffic sign is ”Watch for Children”
The predicted traffic sign is ”Do Not Enter”
The predicted traffic sign is ”Priority Road”
The predicted traffic sign is ”Do Not Enter”
The predicted traffic sign is ”Speed Limit 70”
The predicted traffic sign is ”Watch for Children”
The predicted traffic sign is ”Priority”
The predicted traffic sign is ”Pass by on right”
The predicted traffic sign is ”Road Work”
The predicted traffic sign is ”Roundabout”
The predicted traffic sign is ”Roundabout”
The predicted traffic sign is ”Traffic signals ahead”
The predicted traffic sign is ”Yield”
The predicted traffic sign is ”Yield”

You can see that the Lenet-5 modified model performed better than Lenet-5 model on the images picked up from the web. It correctly classifies 15 out of 16 images that we provide. It correctly classifies 93.75% of the images we provide.